[nodejs] add bun backend #101

cdmistman · 2023-03-30T18:50:54Z

We'd like to have better support for bun and encourage its usage in JS projects. This adds a new backend to UPM for bun, added as the bun language ie upm --language bun.

Note that the bun lockfile format is in a binary format that's only recognizable by bun, but it's also executable, resulting in a yarn lockfile format being printed to stdout. As such, for reading the lockfile, this backend executes the lockfile with bun and then operates on the resulting output.

masad-frost

can you fill in details in the PR please?

internal/backends/nodejs/nodejs.go

internal/backends/backends.go

bun can be used to replace a nodejs packager, but it can also be used in its own runtime. As such, it's not accurate to label it as a 'nodejs' backend

airportyh

Awesome! I am excited.

Jarred-Sumner · 2023-04-06T23:26:28Z

internal/backends/nodejs/nodejs.go

+	Specfile:         "package.json",
+	Lockfile:         "bun.lockb",
+	FilenamePatterns: nodejsPatterns,
+	Quirks: api.QuirksAddRemoveAlsoLocks |


additional env vars that might be helpful:

BUN_CONFIG_SKIP_SAVE_LOCKFILE

BUN_CONFIG_SKIP_LOAD_LOCKFILE

BUN_CONFIG_SKIP_INSTALL_PACKAGES

BUN_INSTALL_VERBOSE

GOMAXPROCS (control max number of threads used, not in go but didn't want to make up a new env var for this)

BUN_MANIFEST_CACHE=0: no cache

BUN_MANIFEST_CACHE=1: read the cache, but ignore Cache-Control headers

BUN_MANIFEST_CACHE=2: read the cache & follow Cache-Control headers (default)

BUN_INSTALL_CACHE_DIR

BUN_INSTALL (directory bun was installed in, not bun install)

also bun pm ls --all could be used to print out the installed package order in node_modules (due to hoisting, will often contain duplicate versions)

There is more config in bunfig.toml: https://github.com/oven-sh/bun/blob/jarred/new-bundler/src/bunfig.zig#L235-L426

due to hoisting, will often contain duplicate versions

That's fine, right now UPM only accepts one version per package, which is problematic anyways. The package version that's last in the output will be the one that's reported by UPM

So there seems to be 2 ways of reading the packages and versions in the bun lockfile:

executing the bun.lockb file

this is what's done by UPM in this PR

is there a way to ensure that the output format is yarn's from the command line, ie without relying on the bunfig.toml file?

via bun pm ls --all

I can probably switch to this, and that would be preferred in my opinion

is the output format considered stable? ie, if I write a regex for reading the package name and version eg (@[a-z]+\/)?[a-z]+@[0-9]+\.[0-9]+\.[0-9]+ is there any possibility that the regex will break in the future?

those env vars are definitely helpful for some other things we have planned with bun, thank you!

bun pm hash-string might be easier for parsing and we cannot change it because doing so would invalidate lockfiles unexpectedly (we use that to determine if we should bother saving the lockfile)

❯ bun pm hash-string -- BEGIN SHA512/256(`${alphabetize(name)}@${order(version)}`) -- @eslint/[email protected] @humanwhocodes/[email protected] @humanwhocodes/[email protected] @humanwhocodes/[email protected] @nodelib/[email protected] @nodelib/[email protected] @nodelib/[email protected] @types/[email protected] @types/[email protected] @types/[email protected] @types/[email protected] @types/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] @typescript-eslint/[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] bun-webkit-linux-amd64@0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26 bun-webkit-linux-arm64@0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26 bun-webkit-macos-amd64@0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26 bun-webkit-macos-arm64@0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26 [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] -- END HASH--

We probably should just add a way to print this info as JSON though.

Also you should set COLORTERM=0 to ensure ansi color codes don't get included in output. It should be smart enough to detect this, but good to proactively disable that

Though bun pm hash-string returns all packages in the lockfile, but not all these packages will ultimately install (bun-webkit-linux-arm64, for example, is not installed on linux x64 but exists in the lockfile so that the lockfile is consistent across operating systems and CPU architectures)

got it, it could be a future feature for UPM to detect platform-specific dependencies and filter/collate them, but that's not a large concern at the moment. Seems bun pm hash-string is the best option here - thanks!

cdmistman · 2023-04-07T22:02:45Z

latest change using bun pm hash-string works on https://github.com/oven-sh/bun - notice tslib is only reported once, which is a limitation in the current design of UPM

$ upm -l bun list -a
name                                    version                                       
-------------------------------------   ----------------------------------------------
@eslint/eslintrc                        1.4.1                                         
@humanwhocodes/config-array             0.11.8                                        
@humanwhocodes/module-importer          1.0.1                                         
@humanwhocodes/object-schema            1.2.1                                         
@nodelib/fs.scandir                     2.1.5                                         
@nodelib/fs.stat                        2.0.5                                         
@nodelib/fs.walk                        1.2.8                                         
@types/json-schema                      7.0.11                                        
@types/prop-types                       15.7.5                                        
@types/react                            18.0.28                                       
@types/scheduler                        0.16.2                                        
@types/semver                           7.3.13                                        
@typescript-eslint/eslint-plugin        5.52.0                                        
@typescript-eslint/parser               5.52.0                                        
@typescript-eslint/scope-manager        5.52.0                                        
@typescript-eslint/type-utils           5.52.0                                        
@typescript-eslint/types                5.52.0                                        
@typescript-eslint/typescript-estree    5.52.0                                        
@typescript-eslint/utils                5.52.0                                        
@typescript-eslint/visitor-keys         5.52.0                                        
acorn                                   8.8.2                                         
acorn-jsx                               5.3.2                                         
ajv                                     6.12.6                                        
ansi-regex                              5.0.1                                         
ansi-styles                             4.3.0                                         
argparse                                2.0.1                                         
array-union                             2.1.0                                         
balanced-match                          1.0.2                                         
brace-expansion                         1.1.11                                        
braces                                  3.0.2                                         
bun-webkit                              0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26
bun-webkit-linux-amd64                  0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26
bun-webkit-linux-arm64                  0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26
bun-webkit-macos-amd64                  0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26
bun-webkit-macos-arm64                  0.0.1-595dd33692e02001631f6c3ab6f2bbb199278d26
callsites                               3.1.0                                         
camel-case                              4.1.2                                         
capital-case                            1.0.4                                         
chalk                                   4.1.2                                         
change-case                             4.1.2                                         
color-convert                           2.0.1                                         
color-name                              1.1.4                                         
concat-map                              0.0.1                                         
constant-case                           3.0.4                                         
cross-spawn                             7.0.3                                         
csstype                                 3.1.1                                         
debug                                   4.3.4                                         
deep-is                                 0.1.4                                         
dir-glob                                3.0.1                                         
doctrine                                3.0.0                                         
dot-case                                3.0.4                                         
escape-string-regexp                    4.0.0                                         
eslint                                  8.34.0                                        
eslint-config-prettier                  8.6.0                                         
eslint-scope                            7.1.1                                         
eslint-utils                            3.0.0                                         
eslint-visitor-keys                     3.3.0                                         
espree                                  9.4.1                                         
esquery                                 1.4.2                                         
esrecurse                               4.3.0                                         
estraverse                              5.3.0                                         
esutils                                 2.0.3                                         
fast-deep-equal                         3.1.3                                         
fast-glob                               3.2.12                                        
fast-json-stable-stringify              2.1.0                                         
fast-levenshtein                        2.0.6                                         
fastq                                   1.15.0                                        
file-entry-cache                        6.0.1                                         
fill-range                              7.0.1                                         
find-up                                 5.0.0                                         
flat-cache                              3.0.4                                         
flatted                                 3.2.7                                         
fs.realpath                             1.0.0                                         
glob                                    7.2.3                                         
glob-parent                             6.0.2                                         
globals                                 13.20.0                                       
globby                                  11.1.0                                        
grapheme-splitter                       1.0.4                                         
has-flag                                4.0.0                                         
header-case                             2.0.4                                         
ignore                                  5.2.4                                         
import-fresh                            3.3.0                                         
imurmurhash                             0.1.4                                         
inflight                                1.0.6                                         
inherits                                2.0.4                                         
is-extglob                              2.1.1                                         
is-glob                                 4.0.3                                         
is-number                               7.0.0                                         
is-path-inside                          3.0.3                                         
isexe                                   2.0.0                                         
js-sdsl                                 4.3.0                                         
js-tokens                               4.0.0                                         
js-yaml                                 4.1.0                                         
json-schema-traverse                    0.4.1                                         
json-stable-stringify-without-jsonify   1.0.1                                         
levn                                    0.4.1                                         
locate-path                             6.0.0                                         
lodash.merge                            4.6.2                                         
loose-envify                            1.4.0                                         
lower-case                              2.0.2                                         
lru-cache                               6.0.0                                         
merge2                                  1.4.1                                         
micromatch                              4.0.5                                         
minimatch                               3.1.2                                         
mitata                                  0.1.6                                         
ms                                      2.1.2                                         
natural-compare                         1.4.0                                         
natural-compare-lite                    1.4.0                                         
no-case                                 3.0.4                                         
once                                    1.4.0                                         
optionator                              0.9.1                                         
p-limit                                 3.1.0                                         
p-locate                                5.0.0                                         
param-case                              3.0.4                                         
parent-module                           1.0.1                                         
pascal-case                             3.1.2                                         
path-case                               3.0.4                                         
path-exists                             4.0.0                                         
path-is-absolute                        1.0.1                                         
path-key                                3.1.1                                         
path-type                               4.0.0                                         
peechy                                  0.4.33                                        
picomatch                               2.3.1                                         
prelude-ls                              1.2.1                                         
prettier                                2.8.4                                         
punycode                                2.3.0                                         
queue-microtask                         1.2.3                                         
react                                   18.3.0-next-4fcc9184a-20230217                
react-dom                               18.3.0-next-4fcc9184a-20230217                
regexpp                                 3.2.0                                         
resolve-from                            4.0.0                                         
reusify                                 1.0.4                                         
rimraf                                  3.0.2                                         
run-parallel                            1.2.0                                         
scheduler                               0.24.0-next-4fcc9184a-20230217                
semver                                  7.3.8                                         
sentence-case                           3.0.4                                         
shebang-command                         2.0.0                                         
shebang-regex                           3.0.0                                         
slash                                   3.0.0                                         
snake-case                              3.0.4                                         
strip-ansi                              6.0.1                                         
strip-json-comments                     3.1.1                                         
supports-color                          7.2.0                                         
text-table                              0.2.0                                         
to-regex-range                          5.0.1                                         
tslib                                   2.5.0                                         
tsutils                                 3.21.0                                        
type-check                              0.4.0                                         
type-fest                               0.20.2                                        
typescript                              5.0.2                                         
upper-case                              2.0.2                                         
upper-case-first                        2.0.2                                         
uri-js                                  4.4.1                                         
which                                   2.0.2                                         
word-wrap                               1.2.3                                         
wrappy                                  1.0.2                                         
yallist                                 4.0.0                                         
yocto-queue                             0.1.0

Jarred-Sumner · 2023-04-07T22:16:24Z

internal/backends/nodejs/nodejs.go

+		// - 1.2.3
+		// - 1.2.3-beta.1
+		// - 1.2.3-beta.1-build.1
+		r := regexp.MustCompile(`(?m)^(@?[^@ \n]+)@([0-9]+\.[0-9]+\.[0-9]+(-.+)?)$`)


When the version is a git dependency, github dependency, a tarball url, a workspace, or a symlink, the format in the version specifier changes. Instead, I suggest matching everything after the last"@" up until the newline character

For example:

ah, good catch - should be fixed in latest commit. thanks!

addressed

cdmistman added 4 commits March 30, 2023 18:45

[nodejs] add bun backend

c0b1a2d

Merge branch 'main' of github.com:replit/upm into cad/bun

caa1586

fix lint

528a3d7

add bun backend to cli

c4789b8

cdmistman marked this pull request as ready for review April 5, 2023 23:01

cdmistman requested review from masad-frost, a team and airportyh and removed request for a team April 5, 2023 23:01

masad-frost previously requested changes Apr 6, 2023

View reviewed changes

airportyh reviewed Apr 6, 2023

View reviewed changes

internal/backends/nodejs/nodejs.go Outdated Show resolved Hide resolved

airportyh reviewed Apr 6, 2023

View reviewed changes

internal/backends/backends.go Outdated Show resolved Hide resolved

nodejs-bun -> bun

20d48da

bun can be used to replace a nodejs packager, but it can also be used in its own runtime. As such, it's not accurate to label it as a 'nodejs' backend

cdmistman requested review from masad-frost and airportyh April 6, 2023 19:47

airportyh approved these changes Apr 6, 2023

View reviewed changes

Jarred-Sumner reviewed Apr 6, 2023

View reviewed changes

cdmistman added 2 commits April 7, 2023 21:35

Revert Yarn Lockfile parsing abstraction

d888c28

use 'bun pm hash-string' for reading bun lockfile

76dddec

Jarred-Sumner reviewed Apr 7, 2023

View reviewed changes

cdmistman added 2 commits April 7, 2023 22:24

fix version matching

2f376dd

cleanup comments

b11e9b1

cdmistman merged commit b7d7b98 into main Apr 10, 2023

cdmistman deleted the cad/bun branch April 10, 2023 21:51

blast-hardcheese added major enhancement New feature or request labels Dec 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[nodejs] add bun backend #101

[nodejs] add bun backend #101

cdmistman commented Mar 30, 2023 •

edited

Loading

masad-frost left a comment

airportyh left a comment

Jarred-Sumner Apr 6, 2023

Jarred-Sumner Apr 6, 2023

Jarred-Sumner Apr 6, 2023

cdmistman Apr 7, 2023 •

edited

Loading

cdmistman Apr 7, 2023

Jarred-Sumner Apr 7, 2023

Jarred-Sumner Apr 7, 2023

cdmistman Apr 7, 2023

cdmistman commented Apr 7, 2023 •

edited

Loading

Jarred-Sumner Apr 7, 2023 •

edited

Loading

cdmistman Apr 7, 2023 •

edited

Loading

[nodejs] add bun backend #101

[nodejs] add bun backend #101

Conversation

cdmistman commented Mar 30, 2023 • edited Loading

masad-frost left a comment

Choose a reason for hiding this comment

airportyh left a comment

Choose a reason for hiding this comment

Jarred-Sumner Apr 6, 2023

Choose a reason for hiding this comment

Jarred-Sumner Apr 6, 2023

Choose a reason for hiding this comment

Jarred-Sumner Apr 6, 2023

Choose a reason for hiding this comment

cdmistman Apr 7, 2023 • edited Loading

Choose a reason for hiding this comment

cdmistman Apr 7, 2023

Choose a reason for hiding this comment

Jarred-Sumner Apr 7, 2023

Choose a reason for hiding this comment

Jarred-Sumner Apr 7, 2023

Choose a reason for hiding this comment

cdmistman Apr 7, 2023

Choose a reason for hiding this comment

cdmistman commented Apr 7, 2023 • edited Loading

Jarred-Sumner Apr 7, 2023 • edited Loading

Choose a reason for hiding this comment

cdmistman Apr 7, 2023 • edited Loading

Choose a reason for hiding this comment

cdmistman commented Mar 30, 2023 •

edited

Loading

cdmistman Apr 7, 2023 •

edited

Loading

cdmistman commented Apr 7, 2023 •

edited

Loading

Jarred-Sumner Apr 7, 2023 •

edited

Loading

cdmistman Apr 7, 2023 •

edited

Loading